This is Indian Bird Species Image Classification.¶

Dataset is from Kaggle. link: https://www.kaggle.com/datasets/arjunbasandrai/25-indian-bird-species-with-226k-images

Will be using Microsoft's ResNet50V2 to fine-tune it according to our dataset.

In [1]:
import tensorflow as tf
tf.__version__
Out[1]:
'2.9.0'
In [41]:
# Import The Libraries 

from tensorflow.keras.layers import Input, Dense, Flatten
from tensorflow.keras.models import Model
from tensorflow.keras.preprocessing import image
from tensorflow.keras.preprocessing.image import ImageDataGenerator, load_img
from tensorflow.keras.models import Sequential
from tensorflow.keras.models import load_model

import numpy as np
from glob import glob
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import os
import random
In [5]:
import sklearn
In [6]:
# Import helper functions - https://gist.github.com/priteshkeleven/5756d60c56b3ac37c86cef348e76452f
from helper_functions import create_tensorboard_callback, plot_loss_curves, unzip_data, walk_through_dir, compare_historys

Getting the dataset¶

In [3]:
# Paths
train_dir = 'training_set'
test_dir = 'testing_set'
In [4]:
IMAGE_SIZE = [224, 224, 3]
In [5]:
# Setup data inputs
import tensorflow as tf

IMG_SIZE = (224, 224)

train_data = tf.keras.preprocessing.image_dataset_from_directory(train_dir,
                                                               image_size=IMG_SIZE,
                                                               label_mode="categorical",
                                                                batch_size=16)

test_data = tf.keras.preprocessing.image_dataset_from_directory(test_dir,
                                                               image_size=IMG_SIZE,
                                                               label_mode="categorical",
                                                               shuffle=False,
                                                               batch_size=16)

Found 16974 files belonging to 25 classes.

2023-03-28 16:23:34.936867: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:34.982309: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:34.982372: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:34.982973: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-28 16:23:34.985405: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:34.985441: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:34.985458: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:36.912465: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:36.912527: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:36.912533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Could not identify NUMA node of platform GPU id 0, defaulting to 0.  Your kernel may not have been built with NUMA support.
2023-03-28 16:23:36.912557: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:23:36.912600: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 3433 MB memory:  -> device: 0, name: NVIDIA GeForce RTX 3060 Laptop GPU, pci bus id: 0000:01:00.0, compute capability: 8.6

Found 5647 files belonging to 25 classes.
In [7]:
# Creat checkpoint callback
checkpoint_path = "25_classes_birds_data_model_checkpoint/checkpoints.ckpt"
checkpoint_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,
                                                        save_weights_only=True,
                                                        monitor="val_accuracy",
                                                        save_best_only=True)

Lets implement data augmentation layer to avoid overfitting¶

In [51]:
# Create data augmentation layer
from tensorflow.keras import layers
from tensorflow.keras import Sequential

data_augmentation = Sequential([
    layers.RandomFlip("horizontal"),
    layers.RandomRotation(0.3),
    layers.RandomHeight(0.3),
    layers.RandomWidth(0.3),
    layers.RandomZoom(0.3),
    layers.Rescaling(1/255.)
], name="data_augmentation")

Visualizing Augmentation layers¶

In [52]:
# View a random image and comapare it to augmentaed image
target_class = random.choice(train_data.class_names)
target_dir = "training_set/" + target_class

random_image = random.choice(os.listdir(target_dir))
random_image_path  = target_dir + "/" + random_image

# Read in the random image
img = mpimg.imread(random_image_path)
plt.imshow(img)
plt.title(f"Original random image from class: {target_class}")
plt.axis(False);

# Plot augmented image
augmented_img = data_augmentation(img, training=True)
plt.figure()
plt.imshow(augmented_img)
plt.title(f"Augmented random image form class: {target_class}")
plt.axis(False);

Using ResNet50V2 as a base pre-trained model¶

In [9]:
base_model = tf.keras.applications.ResNet50V2(include_top=False)
In [10]:
base_model.trainable = False
In [11]:
base_model.summary()

Model: "resnet50v2"
__________________________________________________________________________________________________
 Layer (type)                   Output Shape         Param #     Connected to                     
==================================================================================================
 input_1 (InputLayer)           [(None, None, None,  0           []                               
                                 3)]                                                              
                                                                                                  
 conv1_pad (ZeroPadding2D)      (None, None, None,   0           ['input_1[0][0]']                
                                3)                                                                
                                                                                                  
 conv1_conv (Conv2D)            (None, None, None,   9472        ['conv1_pad[0][0]']              
                                64)                                                               
                                                                                                  
 pool1_pad (ZeroPadding2D)      (None, None, None,   0           ['conv1_conv[0][0]']             
                                64)                                                               
                                                                                                  
 pool1_pool (MaxPooling2D)      (None, None, None,   0           ['pool1_pad[0][0]']              
                                64)                                                               
                                                                                                  
 conv2_block1_preact_bn (BatchN  (None, None, None,   256        ['pool1_pool[0][0]']             
 ormalization)                  64)                                                               
                                                                                                  
 conv2_block1_preact_relu (Acti  (None, None, None,   0          ['conv2_block1_preact_bn[0][0]'] 
 vation)                        64)                                                               
                                                                                                  
 conv2_block1_1_conv (Conv2D)   (None, None, None,   4096        ['conv2_block1_preact_relu[0][0]'
                                64)                              ]                                
                                                                                                  
 conv2_block1_1_bn (BatchNormal  (None, None, None,   256        ['conv2_block1_1_conv[0][0]']    
 ization)                       64)                                                               
                                                                                                  
 conv2_block1_1_relu (Activatio  (None, None, None,   0          ['conv2_block1_1_bn[0][0]']      
 n)                             64)                                                               
                                                                                                  
 conv2_block1_2_pad (ZeroPaddin  (None, None, None,   0          ['conv2_block1_1_relu[0][0]']    
 g2D)                           64)                                                               
                                                                                                  
 conv2_block1_2_conv (Conv2D)   (None, None, None,   36864       ['conv2_block1_2_pad[0][0]']     
                                64)                                                               
                                                                                                  
 conv2_block1_2_bn (BatchNormal  (None, None, None,   256        ['conv2_block1_2_conv[0][0]']    
 ization)                       64)                                                               
                                                                                                  
 conv2_block1_2_relu (Activatio  (None, None, None,   0          ['conv2_block1_2_bn[0][0]']      
 n)                             64)                                                               
                                                                                                  
 conv2_block1_0_conv (Conv2D)   (None, None, None,   16640       ['conv2_block1_preact_relu[0][0]'
                                256)                             ]                                
                                                                                                  
 conv2_block1_3_conv (Conv2D)   (None, None, None,   16640       ['conv2_block1_2_relu[0][0]']    
                                256)                                                              
                                                                                                  
 conv2_block1_out (Add)         (None, None, None,   0           ['conv2_block1_0_conv[0][0]',    
                                256)                              'conv2_block1_3_conv[0][0]']    
                                                                                                  
 conv2_block2_preact_bn (BatchN  (None, None, None,   1024       ['conv2_block1_out[0][0]']       
 ormalization)                  256)                                                              
                                                                                                  
 conv2_block2_preact_relu (Acti  (None, None, None,   0          ['conv2_block2_preact_bn[0][0]'] 
 vation)                        256)                                                              
                                                                                                  
 conv2_block2_1_conv (Conv2D)   (None, None, None,   16384       ['conv2_block2_preact_relu[0][0]'
                                64)                              ]                                
                                                                                                  
 conv2_block2_1_bn (BatchNormal  (None, None, None,   256        ['conv2_block2_1_conv[0][0]']    
 ization)                       64)                                                               
                                                                                                  
 conv2_block2_1_relu (Activatio  (None, None, None,   0          ['conv2_block2_1_bn[0][0]']      
 n)                             64)                                                               
                                                                                                  
 conv2_block2_2_pad (ZeroPaddin  (None, None, None,   0          ['conv2_block2_1_relu[0][0]']    
 g2D)                           64)                                                               
                                                                                                  
 conv2_block2_2_conv (Conv2D)   (None, None, None,   36864       ['conv2_block2_2_pad[0][0]']     
                                64)                                                               
                                                                                                  
 conv2_block2_2_bn (BatchNormal  (None, None, None,   256        ['conv2_block2_2_conv[0][0]']    
 ization)                       64)                                                               
                                                                                                  
 conv2_block2_2_relu (Activatio  (None, None, None,   0          ['conv2_block2_2_bn[0][0]']      
 n)                             64)                                                               
                                                                                                  
 conv2_block2_3_conv (Conv2D)   (None, None, None,   16640       ['conv2_block2_2_relu[0][0]']    
                                256)                                                              
                                                                                                  
 conv2_block2_out (Add)         (None, None, None,   0           ['conv2_block1_out[0][0]',       
                                256)                              'conv2_block2_3_conv[0][0]']    
                                                                                                  
 conv2_block3_preact_bn (BatchN  (None, None, None,   1024       ['conv2_block2_out[0][0]']       
 ormalization)                  256)                                                              
                                                                                                  
 conv2_block3_preact_relu (Acti  (None, None, None,   0          ['conv2_block3_preact_bn[0][0]'] 
 vation)                        256)                                                              
                                                                                                  
 conv2_block3_1_conv (Conv2D)   (None, None, None,   16384       ['conv2_block3_preact_relu[0][0]'
                                64)                              ]                                
                                                                                                  
 conv2_block3_1_bn (BatchNormal  (None, None, None,   256        ['conv2_block3_1_conv[0][0]']    
 ization)                       64)                                                               
                                                                                                  
 conv2_block3_1_relu (Activatio  (None, None, None,   0          ['conv2_block3_1_bn[0][0]']      
 n)                             64)                                                               
                                                                                                  
 conv2_block3_2_pad (ZeroPaddin  (None, None, None,   0          ['conv2_block3_1_relu[0][0]']    
 g2D)                           64)                                                               
                                                                                                  
 conv2_block3_2_conv (Conv2D)   (None, None, None,   36864       ['conv2_block3_2_pad[0][0]']     
                                64)                                                               
                                                                                                  
 conv2_block3_2_bn (BatchNormal  (None, None, None,   256        ['conv2_block3_2_conv[0][0]']    
 ization)                       64)                                                               
                                                                                                  
 conv2_block3_2_relu (Activatio  (None, None, None,   0          ['conv2_block3_2_bn[0][0]']      
 n)                             64)                                                               
                                                                                                  
 max_pooling2d (MaxPooling2D)   (None, None, None,   0           ['conv2_block2_out[0][0]']       
                                256)                                                              
                                                                                                  
 conv2_block3_3_conv (Conv2D)   (None, None, None,   16640       ['conv2_block3_2_relu[0][0]']    
                                256)                                                              
                                                                                                  
 conv2_block3_out (Add)         (None, None, None,   0           ['max_pooling2d[0][0]',          
                                256)                              'conv2_block3_3_conv[0][0]']    
                                                                                                  
 conv3_block1_preact_bn (BatchN  (None, None, None,   1024       ['conv2_block3_out[0][0]']       
 ormalization)                  256)                                                              
                                                                                                  
 conv3_block1_preact_relu (Acti  (None, None, None,   0          ['conv3_block1_preact_bn[0][0]'] 
 vation)                        256)                                                              
                                                                                                  
 conv3_block1_1_conv (Conv2D)   (None, None, None,   32768       ['conv3_block1_preact_relu[0][0]'
                                128)                             ]                                
                                                                                                  
 conv3_block1_1_bn (BatchNormal  (None, None, None,   512        ['conv3_block1_1_conv[0][0]']    
 ization)                       128)                                                              
                                                                                                  
 conv3_block1_1_relu (Activatio  (None, None, None,   0          ['conv3_block1_1_bn[0][0]']      
 n)                             128)                                                              
                                                                                                  
 conv3_block1_2_pad (ZeroPaddin  (None, None, None,   0          ['conv3_block1_1_relu[0][0]']    
 g2D)                           128)                                                              
                                                                                                  
 conv3_block1_2_conv (Conv2D)   (None, None, None,   147456      ['conv3_block1_2_pad[0][0]']     
                                128)                                                              
                                                                                                  
 conv3_block1_2_bn (BatchNormal  (None, None, None,   512        ['conv3_block1_2_conv[0][0]']    
 ization)                       128)                                                              
                                                                                                  
 conv3_block1_2_relu (Activatio  (None, None, None,   0          ['conv3_block1_2_bn[0][0]']      
 n)                             128)                                                              
                                                                                                  
 conv3_block1_0_conv (Conv2D)   (None, None, None,   131584      ['conv3_block1_preact_relu[0][0]'
                                512)                             ]                                
                                                                                                  
 conv3_block1_3_conv (Conv2D)   (None, None, None,   66048       ['conv3_block1_2_relu[0][0]']    
                                512)                                                              
                                                                                                  
 conv3_block1_out (Add)         (None, None, None,   0           ['conv3_block1_0_conv[0][0]',    
                                512)                              'conv3_block1_3_conv[0][0]']    
                                                                                                  
 conv3_block2_preact_bn (BatchN  (None, None, None,   2048       ['conv3_block1_out[0][0]']       
 ormalization)                  512)                                                              
                                                                                                  
 conv3_block2_preact_relu (Acti  (None, None, None,   0          ['conv3_block2_preact_bn[0][0]'] 
 vation)                        512)                                                              
                                                                                                  
 conv3_block2_1_conv (Conv2D)   (None, None, None,   65536       ['conv3_block2_preact_relu[0][0]'
                                128)                             ]                                
                                                                                                  
 conv3_block2_1_bn (BatchNormal  (None, None, None,   512        ['conv3_block2_1_conv[0][0]']    
 ization)                       128)                                                              
                                                                                                  
 conv3_block2_1_relu (Activatio  (None, None, None,   0          ['conv3_block2_1_bn[0][0]']      
 n)                             128)                                                              
                                                                                                  
 conv3_block2_2_pad (ZeroPaddin  (None, None, None,   0          ['conv3_block2_1_relu[0][0]']    
 g2D)                           128)                                                              
                                                                                                  
 conv3_block2_2_conv (Conv2D)   (None, None, None,   147456      ['conv3_block2_2_pad[0][0]']     
                                128)                                                              
                                                                                                  
 conv3_block2_2_bn (BatchNormal  (None, None, None,   512        ['conv3_block2_2_conv[0][0]']    
 ization)                       128)                                                              
                                                                                                  
 conv3_block2_2_relu (Activatio  (None, None, None,   0          ['conv3_block2_2_bn[0][0]']      
 n)                             128)                                                              
                                                                                                  
 conv3_block2_3_conv (Conv2D)   (None, None, None,   66048       ['conv3_block2_2_relu[0][0]']    
                                512)                                                              
                                                                                                  
 conv3_block2_out (Add)         (None, None, None,   0           ['conv3_block1_out[0][0]',       
                                512)                              'conv3_block2_3_conv[0][0]']    
                                                                                                  
 conv3_block3_preact_bn (BatchN  (None, None, None,   2048       ['conv3_block2_out[0][0]']       
 ormalization)                  512)                                                              
                                                                                                  
 conv3_block3_preact_relu (Acti  (None, None, None,   0          ['conv3_block3_preact_bn[0][0]'] 
 vation)                        512)                                                              
                                                                                                  
 conv3_block3_1_conv (Conv2D)   (None, None, None,   65536       ['conv3_block3_preact_relu[0][0]'
                                128)                             ]                                
                                                                                                  
 conv3_block3_1_bn (BatchNormal  (None, None, None,   512        ['conv3_block3_1_conv[0][0]']    
 ization)                       128)                                                              
                                                                                                  
 conv3_block3_1_relu (Activatio  (None, None, None,   0          ['conv3_block3_1_bn[0][0]']      
 n)                             128)                                                              
                                                                                                  
 conv3_block3_2_pad (ZeroPaddin  (None, None, None,   0          ['conv3_block3_1_relu[0][0]']    
 g2D)                           128)                                                              
                                                                                                  
 conv3_block3_2_conv (Conv2D)   (None, None, None,   147456      ['conv3_block3_2_pad[0][0]']     
                                128)                                                              
                                                                                                  
 conv3_block3_2_bn (BatchNormal  (None, None, None,   512        ['conv3_block3_2_conv[0][0]']    
 ization)                       128)                                                              
                                                                                                  
 conv3_block3_2_relu (Activatio  (None, None, None,   0          ['conv3_block3_2_bn[0][0]']      
 n)                             128)                                                              
                                                                                                  
 conv3_block3_3_conv (Conv2D)   (None, None, None,   66048       ['conv3_block3_2_relu[0][0]']    
                                512)                                                              
                                                                                                  
 conv3_block3_out (Add)         (None, None, None,   0           ['conv3_block2_out[0][0]',       
                                512)                              'conv3_block3_3_conv[0][0]']    
                                                                                                  
 conv3_block4_preact_bn (BatchN  (None, None, None,   2048       ['conv3_block3_out[0][0]']       
 ormalization)                  512)                                                              
                                                                                                  
 conv3_block4_preact_relu (Acti  (None, None, None,   0          ['conv3_block4_preact_bn[0][0]'] 
 vation)                        512)                                                              
                                                                                                  
 conv3_block4_1_conv (Conv2D)   (None, None, None,   65536       ['conv3_block4_preact_relu[0][0]'
                                128)                             ]                                
                                                                                                  
 conv3_block4_1_bn (BatchNormal  (None, None, None,   512        ['conv3_block4_1_conv[0][0]']    
 ization)                       128)                                                              
                                                                                                  
 conv3_block4_1_relu (Activatio  (None, None, None,   0          ['conv3_block4_1_bn[0][0]']      
 n)                             128)                                                              
                                                                                                  
 conv3_block4_2_pad (ZeroPaddin  (None, None, None,   0          ['conv3_block4_1_relu[0][0]']    
 g2D)                           128)                                                              
                                                                                                  
 conv3_block4_2_conv (Conv2D)   (None, None, None,   147456      ['conv3_block4_2_pad[0][0]']     
                                128)                                                              
                                                                                                  
 conv3_block4_2_bn (BatchNormal  (None, None, None,   512        ['conv3_block4_2_conv[0][0]']    
 ization)                       128)                                                              
                                                                                                  
 conv3_block4_2_relu (Activatio  (None, None, None,   0          ['conv3_block4_2_bn[0][0]']      
 n)                             128)                                                              
                                                                                                  
 max_pooling2d_1 (MaxPooling2D)  (None, None, None,   0          ['conv3_block3_out[0][0]']       
                                512)                                                              
                                                                                                  
 conv3_block4_3_conv (Conv2D)   (None, None, None,   66048       ['conv3_block4_2_relu[0][0]']    
                                512)                                                              
                                                                                                  
 conv3_block4_out (Add)         (None, None, None,   0           ['max_pooling2d_1[0][0]',        
                                512)                              'conv3_block4_3_conv[0][0]']    
                                                                                                  
 conv4_block1_preact_bn (BatchN  (None, None, None,   2048       ['conv3_block4_out[0][0]']       
 ormalization)                  512)                                                              
                                                                                                  
 conv4_block1_preact_relu (Acti  (None, None, None,   0          ['conv4_block1_preact_bn[0][0]'] 
 vation)                        512)                                                              
                                                                                                  
 conv4_block1_1_conv (Conv2D)   (None, None, None,   131072      ['conv4_block1_preact_relu[0][0]'
                                256)                             ]                                
                                                                                                  
 conv4_block1_1_bn (BatchNormal  (None, None, None,   1024       ['conv4_block1_1_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block1_1_relu (Activatio  (None, None, None,   0          ['conv4_block1_1_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block1_2_pad (ZeroPaddin  (None, None, None,   0          ['conv4_block1_1_relu[0][0]']    
 g2D)                           256)                                                              
                                                                                                  
 conv4_block1_2_conv (Conv2D)   (None, None, None,   589824      ['conv4_block1_2_pad[0][0]']     
                                256)                                                              
                                                                                                  
 conv4_block1_2_bn (BatchNormal  (None, None, None,   1024       ['conv4_block1_2_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block1_2_relu (Activatio  (None, None, None,   0          ['conv4_block1_2_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block1_0_conv (Conv2D)   (None, None, None,   525312      ['conv4_block1_preact_relu[0][0]'
                                1024)                            ]                                
                                                                                                  
 conv4_block1_3_conv (Conv2D)   (None, None, None,   263168      ['conv4_block1_2_relu[0][0]']    
                                1024)                                                             
                                                                                                  
 conv4_block1_out (Add)         (None, None, None,   0           ['conv4_block1_0_conv[0][0]',    
                                1024)                             'conv4_block1_3_conv[0][0]']    
                                                                                                  
 conv4_block2_preact_bn (BatchN  (None, None, None,   4096       ['conv4_block1_out[0][0]']       
 ormalization)                  1024)                                                             
                                                                                                  
 conv4_block2_preact_relu (Acti  (None, None, None,   0          ['conv4_block2_preact_bn[0][0]'] 
 vation)                        1024)                                                             
                                                                                                  
 conv4_block2_1_conv (Conv2D)   (None, None, None,   262144      ['conv4_block2_preact_relu[0][0]'
                                256)                             ]                                
                                                                                                  
 conv4_block2_1_bn (BatchNormal  (None, None, None,   1024       ['conv4_block2_1_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block2_1_relu (Activatio  (None, None, None,   0          ['conv4_block2_1_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block2_2_pad (ZeroPaddin  (None, None, None,   0          ['conv4_block2_1_relu[0][0]']    
 g2D)                           256)                                                              
                                                                                                  
 conv4_block2_2_conv (Conv2D)   (None, None, None,   589824      ['conv4_block2_2_pad[0][0]']     
                                256)                                                              
                                                                                                  
 conv4_block2_2_bn (BatchNormal  (None, None, None,   1024       ['conv4_block2_2_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block2_2_relu (Activatio  (None, None, None,   0          ['conv4_block2_2_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block2_3_conv (Conv2D)   (None, None, None,   263168      ['conv4_block2_2_relu[0][0]']    
                                1024)                                                             
                                                                                                  
 conv4_block2_out (Add)         (None, None, None,   0           ['conv4_block1_out[0][0]',       
                                1024)                             'conv4_block2_3_conv[0][0]']    
                                                                                                  
 conv4_block3_preact_bn (BatchN  (None, None, None,   4096       ['conv4_block2_out[0][0]']       
 ormalization)                  1024)                                                             
                                                                                                  
 conv4_block3_preact_relu (Acti  (None, None, None,   0          ['conv4_block3_preact_bn[0][0]'] 
 vation)                        1024)                                                             
                                                                                                  
 conv4_block3_1_conv (Conv2D)   (None, None, None,   262144      ['conv4_block3_preact_relu[0][0]'
                                256)                             ]                                
                                                                                                  
 conv4_block3_1_bn (BatchNormal  (None, None, None,   1024       ['conv4_block3_1_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block3_1_relu (Activatio  (None, None, None,   0          ['conv4_block3_1_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block3_2_pad (ZeroPaddin  (None, None, None,   0          ['conv4_block3_1_relu[0][0]']    
 g2D)                           256)                                                              
                                                                                                  
 conv4_block3_2_conv (Conv2D)   (None, None, None,   589824      ['conv4_block3_2_pad[0][0]']     
                                256)                                                              
                                                                                                  
 conv4_block3_2_bn (BatchNormal  (None, None, None,   1024       ['conv4_block3_2_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block3_2_relu (Activatio  (None, None, None,   0          ['conv4_block3_2_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block3_3_conv (Conv2D)   (None, None, None,   263168      ['conv4_block3_2_relu[0][0]']    
                                1024)                                                             
                                                                                                  
 conv4_block3_out (Add)         (None, None, None,   0           ['conv4_block2_out[0][0]',       
                                1024)                             'conv4_block3_3_conv[0][0]']    
                                                                                                  
 conv4_block4_preact_bn (BatchN  (None, None, None,   4096       ['conv4_block3_out[0][0]']       
 ormalization)                  1024)                                                             
                                                                                                  
 conv4_block4_preact_relu (Acti  (None, None, None,   0          ['conv4_block4_preact_bn[0][0]'] 
 vation)                        1024)                                                             
                                                                                                  
 conv4_block4_1_conv (Conv2D)   (None, None, None,   262144      ['conv4_block4_preact_relu[0][0]'
                                256)                             ]                                
                                                                                                  
 conv4_block4_1_bn (BatchNormal  (None, None, None,   1024       ['conv4_block4_1_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block4_1_relu (Activatio  (None, None, None,   0          ['conv4_block4_1_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block4_2_pad (ZeroPaddin  (None, None, None,   0          ['conv4_block4_1_relu[0][0]']    
 g2D)                           256)                                                              
                                                                                                  
 conv4_block4_2_conv (Conv2D)   (None, None, None,   589824      ['conv4_block4_2_pad[0][0]']     
                                256)                                                              
                                                                                                  
 conv4_block4_2_bn (BatchNormal  (None, None, None,   1024       ['conv4_block4_2_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block4_2_relu (Activatio  (None, None, None,   0          ['conv4_block4_2_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block4_3_conv (Conv2D)   (None, None, None,   263168      ['conv4_block4_2_relu[0][0]']    
                                1024)                                                             
                                                                                                  
 conv4_block4_out (Add)         (None, None, None,   0           ['conv4_block3_out[0][0]',       
                                1024)                             'conv4_block4_3_conv[0][0]']    
                                                                                                  
 conv4_block5_preact_bn (BatchN  (None, None, None,   4096       ['conv4_block4_out[0][0]']       
 ormalization)                  1024)                                                             
                                                                                                  
 conv4_block5_preact_relu (Acti  (None, None, None,   0          ['conv4_block5_preact_bn[0][0]'] 
 vation)                        1024)                                                             
                                                                                                  
 conv4_block5_1_conv (Conv2D)   (None, None, None,   262144      ['conv4_block5_preact_relu[0][0]'
                                256)                             ]                                
                                                                                                  
 conv4_block5_1_bn (BatchNormal  (None, None, None,   1024       ['conv4_block5_1_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block5_1_relu (Activatio  (None, None, None,   0          ['conv4_block5_1_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block5_2_pad (ZeroPaddin  (None, None, None,   0          ['conv4_block5_1_relu[0][0]']    
 g2D)                           256)                                                              
                                                                                                  
 conv4_block5_2_conv (Conv2D)   (None, None, None,   589824      ['conv4_block5_2_pad[0][0]']     
                                256)                                                              
                                                                                                  
 conv4_block5_2_bn (BatchNormal  (None, None, None,   1024       ['conv4_block5_2_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block5_2_relu (Activatio  (None, None, None,   0          ['conv4_block5_2_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block5_3_conv (Conv2D)   (None, None, None,   263168      ['conv4_block5_2_relu[0][0]']    
                                1024)                                                             
                                                                                                  
 conv4_block5_out (Add)         (None, None, None,   0           ['conv4_block4_out[0][0]',       
                                1024)                             'conv4_block5_3_conv[0][0]']    
                                                                                                  
 conv4_block6_preact_bn (BatchN  (None, None, None,   4096       ['conv4_block5_out[0][0]']       
 ormalization)                  1024)                                                             
                                                                                                  
 conv4_block6_preact_relu (Acti  (None, None, None,   0          ['conv4_block6_preact_bn[0][0]'] 
 vation)                        1024)                                                             
                                                                                                  
 conv4_block6_1_conv (Conv2D)   (None, None, None,   262144      ['conv4_block6_preact_relu[0][0]'
                                256)                             ]                                
                                                                                                  
 conv4_block6_1_bn (BatchNormal  (None, None, None,   1024       ['conv4_block6_1_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block6_1_relu (Activatio  (None, None, None,   0          ['conv4_block6_1_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 conv4_block6_2_pad (ZeroPaddin  (None, None, None,   0          ['conv4_block6_1_relu[0][0]']    
 g2D)                           256)                                                              
                                                                                                  
 conv4_block6_2_conv (Conv2D)   (None, None, None,   589824      ['conv4_block6_2_pad[0][0]']     
                                256)                                                              
                                                                                                  
 conv4_block6_2_bn (BatchNormal  (None, None, None,   1024       ['conv4_block6_2_conv[0][0]']    
 ization)                       256)                                                              
                                                                                                  
 conv4_block6_2_relu (Activatio  (None, None, None,   0          ['conv4_block6_2_bn[0][0]']      
 n)                             256)                                                              
                                                                                                  
 max_pooling2d_2 (MaxPooling2D)  (None, None, None,   0          ['conv4_block5_out[0][0]']       
                                1024)                                                             
                                                                                                  
 conv4_block6_3_conv (Conv2D)   (None, None, None,   263168      ['conv4_block6_2_relu[0][0]']    
                                1024)                                                             
                                                                                                  
 conv4_block6_out (Add)         (None, None, None,   0           ['max_pooling2d_2[0][0]',        
                                1024)                             'conv4_block6_3_conv[0][0]']    
                                                                                                  
 conv5_block1_preact_bn (BatchN  (None, None, None,   4096       ['conv4_block6_out[0][0]']       
 ormalization)                  1024)                                                             
                                                                                                  
 conv5_block1_preact_relu (Acti  (None, None, None,   0          ['conv5_block1_preact_bn[0][0]'] 
 vation)                        1024)                                                             
                                                                                                  
 conv5_block1_1_conv (Conv2D)   (None, None, None,   524288      ['conv5_block1_preact_relu[0][0]'
                                512)                             ]                                
                                                                                                  
 conv5_block1_1_bn (BatchNormal  (None, None, None,   2048       ['conv5_block1_1_conv[0][0]']    
 ization)                       512)                                                              
                                                                                                  
 conv5_block1_1_relu (Activatio  (None, None, None,   0          ['conv5_block1_1_bn[0][0]']      
 n)                             512)                                                              
                                                                                                  
 conv5_block1_2_pad (ZeroPaddin  (None, None, None,   0          ['conv5_block1_1_relu[0][0]']    
 g2D)                           512)                                                              
                                                                                                  
 conv5_block1_2_conv (Conv2D)   (None, None, None,   2359296     ['conv5_block1_2_pad[0][0]']     
                                512)                                                              
                                                                                                  
 conv5_block1_2_bn (BatchNormal  (None, None, None,   2048       ['conv5_block1_2_conv[0][0]']    
 ization)                       512)                                                              
                                                                                                  
 conv5_block1_2_relu (Activatio  (None, None, None,   0          ['conv5_block1_2_bn[0][0]']      
 n)                             512)                                                              
                                                                                                  
 conv5_block1_0_conv (Conv2D)   (None, None, None,   2099200     ['conv5_block1_preact_relu[0][0]'
                                2048)                            ]                                
                                                                                                  
 conv5_block1_3_conv (Conv2D)   (None, None, None,   1050624     ['conv5_block1_2_relu[0][0]']    
                                2048)                                                             
                                                                                                  
 conv5_block1_out (Add)         (None, None, None,   0           ['conv5_block1_0_conv[0][0]',    
                                2048)                             'conv5_block1_3_conv[0][0]']    
                                                                                                  
 conv5_block2_preact_bn (BatchN  (None, None, None,   8192       ['conv5_block1_out[0][0]']       
 ormalization)                  2048)                                                             
                                                                                                  
 conv5_block2_preact_relu (Acti  (None, None, None,   0          ['conv5_block2_preact_bn[0][0]'] 
 vation)                        2048)                                                             
                                                                                                  
 conv5_block2_1_conv (Conv2D)   (None, None, None,   1048576     ['conv5_block2_preact_relu[0][0]'
                                512)                             ]                                
                                                                                                  
 conv5_block2_1_bn (BatchNormal  (None, None, None,   2048       ['conv5_block2_1_conv[0][0]']    
 ization)                       512)                                                              
                                                                                                  
 conv5_block2_1_relu (Activatio  (None, None, None,   0          ['conv5_block2_1_bn[0][0]']      
 n)                             512)                                                              
                                                                                                  
 conv5_block2_2_pad (ZeroPaddin  (None, None, None,   0          ['conv5_block2_1_relu[0][0]']    
 g2D)                           512)                                                              
                                                                                                  
 conv5_block2_2_conv (Conv2D)   (None, None, None,   2359296     ['conv5_block2_2_pad[0][0]']     
                                512)                                                              
                                                                                                  
 conv5_block2_2_bn (BatchNormal  (None, None, None,   2048       ['conv5_block2_2_conv[0][0]']    
 ization)                       512)                                                              
                                                                                                  
 conv5_block2_2_relu (Activatio  (None, None, None,   0          ['conv5_block2_2_bn[0][0]']      
 n)                             512)                                                              
                                                                                                  
 conv5_block2_3_conv (Conv2D)   (None, None, None,   1050624     ['conv5_block2_2_relu[0][0]']    
                                2048)                                                             
                                                                                                  
 conv5_block2_out (Add)         (None, None, None,   0           ['conv5_block1_out[0][0]',       
                                2048)                             'conv5_block2_3_conv[0][0]']    
                                                                                                  
 conv5_block3_preact_bn (BatchN  (None, None, None,   8192       ['conv5_block2_out[0][0]']       
 ormalization)                  2048)                                                             
                                                                                                  
 conv5_block3_preact_relu (Acti  (None, None, None,   0          ['conv5_block3_preact_bn[0][0]'] 
 vation)                        2048)                                                             
                                                                                                  
 conv5_block3_1_conv (Conv2D)   (None, None, None,   1048576     ['conv5_block3_preact_relu[0][0]'
                                512)                             ]                                
                                                                                                  
 conv5_block3_1_bn (BatchNormal  (None, None, None,   2048       ['conv5_block3_1_conv[0][0]']    
 ization)                       512)                                                              
                                                                                                  
 conv5_block3_1_relu (Activatio  (None, None, None,   0          ['conv5_block3_1_bn[0][0]']      
 n)                             512)                                                              
                                                                                                  
 conv5_block3_2_pad (ZeroPaddin  (None, None, None,   0          ['conv5_block3_1_relu[0][0]']    
 g2D)                           512)                                                              
                                                                                                  
 conv5_block3_2_conv (Conv2D)   (None, None, None,   2359296     ['conv5_block3_2_pad[0][0]']     
                                512)                                                              
                                                                                                  
 conv5_block3_2_bn (BatchNormal  (None, None, None,   2048       ['conv5_block3_2_conv[0][0]']    
 ization)                       512)                                                              
                                                                                                  
 conv5_block3_2_relu (Activatio  (None, None, None,   0          ['conv5_block3_2_bn[0][0]']      
 n)                             512)                                                              
                                                                                                  
 conv5_block3_3_conv (Conv2D)   (None, None, None,   1050624     ['conv5_block3_2_relu[0][0]']    
                                2048)                                                             
                                                                                                  
 conv5_block3_out (Add)         (None, None, None,   0           ['conv5_block2_out[0][0]',       
                                2048)                             'conv5_block3_3_conv[0][0]']    
                                                                                                  
 post_bn (BatchNormalization)   (None, None, None,   8192        ['conv5_block3_out[0][0]']       
                                2048)                                                             
                                                                                                  
 post_relu (Activation)         (None, None, None,   0           ['post_bn[0][0]']                
                                2048)                                                             
                                                                                                  
==================================================================================================
Total params: 23,564,800
Trainable params: 0
Non-trainable params: 23,564,800
__________________________________________________________________________________________________

Structuring base_model¶

In [12]:
# Setup model architecture with trainable top layres
inputs = layers.Input(shape=(224, 224, 3), name="input_layer")
x = data_augmentation(inputs) # Augment images during training
x = base_model(x, training=False) # put base model in inference mode to set weights frozen
x = layers.GlobalAveragePooling2D(name="global_avg_pool_layer")(x)
outputs = layers.Dense(len(train_data.class_names), activation="softmax", name="output_layer")(x)

model = tf.keras.Model(inputs, outputs)
In [13]:
# Compile the model
model.compile(loss="categorical_crossentropy",
             optimizer=tf.keras.optimizers.Adam(),
             metrics=["accuracy"])
In [14]:
model.summary()

Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_layer (InputLayer)    [(None, 224, 224, 3)]     0         
                                                                 
 data_augmentation (Sequenti  (None, 224, 224, 3)      0         
 al)                                                             
                                                                 
 resnet50v2 (Functional)     (None, None, None, 2048)  23564800  
                                                                 
 global_avg_pool_layer (Glob  (None, 2048)             0         
 alAveragePooling2D)                                             
                                                                 
 output_layer (Dense)        (None, 25)                51225     
                                                                 
=================================================================
Total params: 23,616,025
Trainable params: 51,225
Non-trainable params: 23,564,800
_________________________________________________________________

Training base model with our dataset¶

In [ ]:
# Fit the model
base_model_history = model.fit(train_data,
                              epochs=5,
                              validation_data=test_data,
                              validation_steps=int(0.15*len(test_data)),
                              callbacks=[checkpoint_callback, create_tensorboard_callback(dir_name="transfer_learning", experiment_name="base_model")])

Load already saved model¶

In [16]:
model = tf.keras.models.load_model('bird_clf_resnet50v2_base_model.h5')
In [31]:
# Evaluate on whole test dataset
base_model_results = model.evaluate(test_data)
base_model_results

239/353 [===================>..........] - ETA: 4s - loss: 0.6411 - accuracy: 0.8324
2023-03-25 22:07:56.981868: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

250/353 [====================>.........] - ETA: 4s - loss: 0.6377 - accuracy: 0.8315
2023-03-25 22:07:57.382174: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-25 22:07:57.382757: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

317/353 [=========================>....] - ETA: 1s - loss: 0.5826 - accuracy: 0.8452
2023-03-25 22:07:59.938045: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

353/353 [==============================] - 15s 41ms/step - loss: 0.5703 - accuracy: 0.8505
Out[31]:
[0.5703216195106506, 0.8505401015281677]
In [32]:
plot_loss_curves(base_model_history)
In [34]:
model.save('bird_clf_resnet50v2_base_model.h5')

Fine Tuning - Last 5 Layers¶

In [35]:
# unfreeze all of the layers in the base model
base_model.trainable = True

# Refreeze every layer except last 5
for layer in base_model.layers[:-5]:
    layer.trainable = False
In [36]:
for i, layer in enumerate(base_model.layers[-10:]):
    print(i, layer.name, layer.trainable)
    

0 conv5_block3_1_bn False
1 conv5_block3_1_relu False
2 conv5_block3_2_pad False
3 conv5_block3_2_conv False
4 conv5_block3_2_bn False
5 conv5_block3_2_relu True
6 conv5_block3_3_conv True
7 conv5_block3_out True
8 post_bn True
9 post_relu True
In [37]:
# Recompiling model with lower learning_rate
model.compile(loss="categorical_crossentropy",
             optimizer=tf.keras.optimizers.Adam(learning_rate=0.0001),
             metrics=["accuracy"])
In [38]:
model.summary()

Model: "model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_layer (InputLayer)    [(None, 224, 224, 3)]     0         
                                                                 
 data_augmentation (Sequenti  (None, 224, 224, 3)      0         
 al)                                                             
                                                                 
 resnet50v2 (Functional)     (None, None, None, 2048)  23564800  
                                                                 
 global_avg_pool_layer (Glob  (None, 2048)             0         
 alAveragePooling2D)                                             
                                                                 
 output_layer (Dense)        (None, 25)                51225     
                                                                 
=================================================================
Total params: 23,616,025
Trainable params: 1,105,945
Non-trainable params: 22,510,080
_________________________________________________________________
In [ ]:
# Fit the model again
initial_epochs = 5
history_fine_tuned_5_layers_base_model = model.fit(train_data,
                                                 epochs=initial_epochs + 5,
                                                 validation_data=test_data,
                                                 validation_steps=int(0.15 * len(test_data)),
                                                 initial_epoch=base_model_history.epoch[-1],
                                                 callbacks=[create_tensorboard_callback(dir_name="transfer_learning",
                                                                                       experiment_name="history_fine_tuned_5_layers_base_model")])
In [40]:
# Evaluate on whole test dataset
fine_tune_5_layers_results = model.evaluate(test_data)
fine_tune_5_layers_results

238/353 [===================>..........] - ETA: 4s - loss: 0.5325 - accuracy: 0.8564
2023-03-25 22:42:12.235521: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

250/353 [====================>.........] - ETA: 4s - loss: 0.5237 - accuracy: 0.8580
2023-03-25 22:42:12.702094: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-25 22:42:12.703044: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

318/353 [==========================>...] - ETA: 1s - loss: 0.4868 - accuracy: 0.8667
2023-03-25 22:42:15.542373: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

353/353 [==============================] - 15s 42ms/step - loss: 0.4837 - accuracy: 0.8697
Out[40]:
[0.4836820960044861, 0.8696653246879578]

Load fine-tuned model¶

In [118]:
model = tf.keras.models.load_model('bird_clf_resnet50v2_fine_tuned_5_layers.h5')
In [41]:
compare_historys(original_history=base_model_history,
                new_history=history_fine_tuned_5_layers_base_model,
                initial_epochs=5)
In [6]:
model = tf.keras.models.load_model("bird_clf_resnet50v2_fine_tuned_5_layers.h5")

Save Model¶

In [7]:
# Save it as a h5 file
model.save('bird_clf_resnet50v2_fine_tuned_5_layers.h5')
In [8]:
from rl.agents.dqn import DQNAgent
from rl.policy import LinearAnnealedPolicy, BoltzmannQPolicy, EpsGreedyQPolicy
from rl.memory import SequentialMemory
from rl.core import Processor

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[8], line 1
----> 1 from rl.agents.dqn import DQNAgent
      2 from rl.policy import LinearAnnealedPolicy, BoltzmannQPolicy, EpsGreedyQPolicy
      3 from rl.memory import SequentialMemory

ModuleNotFoundError: No module named 'rl.agents'
In [121]:
import tensorflow as tf
from tensorflow.keras.layers import Input, Dense, Flatten
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
import gym
import os
In [122]:
INPUT_SHAPE = (224, 224, 3)
In [123]:
class ClassifyProcessor(Processor):
    def process_observation(self, observation):
        img = observation.reshape(INPUT_SHAPE)
        processed_observation = np.array(img)
        return processed_observation

    def process_state_batch(self, batch):
        batch = batch.reshape((-1,) + INPUT_SHAPE)
        processed_batch = batch.astype('float32') / 1.
        return processed_batch

    def process_reward(self, reward):
        return np.clip(reward, -1., 1.)
In [124]:
memory = SequentialMemory(limit=100000, window_length=1)
processor = ClassifyProcessor()
policy = LinearAnnealedPolicy(EpsGreedyQPolicy(), attr='eps', value_max=1., value_min=.1, value_test=.05,
                              nb_steps=100000)
In [126]:
model.summary()

Model: "model_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 input_layer (InputLayer)    [(None, 224, 224, 3)]     0         
                                                                 
 data_augmentation (Sequenti  (None, 224, 224, 3)      0         
 al)                                                             
                                                                 
 resnet50v2 (Functional)     (None, None, None, 2048)  23564800  
                                                                 
 global_avg_pool_layer (Glob  (None, 2048)             0         
 alAveragePooling2D)                                             
                                                                 
 output_layer (Dense)        (None, 25)                51225     
                                                                 
=================================================================
Total params: 23,616,025
Trainable params: 1,105,945
Non-trainable params: 22,510,080
_________________________________________________________________
In [127]:
dqn = DQNAgent(model=model, nb_actions=len(test_data.class_names), policy=policy, memory=memory,
               processor=processor, nb_steps_warmup=50000, gamma=0.5, target_model_update=10000,
               train_interval=4, delta_clip=1.)
dqn.compile(Adam(learning_rate=.00025), metrics=['mae'])

dqn.fit(env, nb_steps=training_steps, log_interval=60000)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[127], line 1
----> 1 dqn = DQNAgent(model=model, nb_actions=len(test_data.class_names), policy=policy, memory=memory,
      2                processor=processor, nb_steps_warmup=50000, gamma=0.5, target_model_update=10000,
      3                train_interval=4, delta_clip=1.)
      4 dqn.compile(Adam(learning_rate=.00025), metrics=['mae'])
      6 dqn.fit(env, nb_steps=training_steps, log_interval=60000)

File ~/miniconda3/envs/hf/lib/python3.10/site-packages/rl/agents/dqn.py:108, in DQNAgent.__init__(self, model, policy, test_policy, enable_double_dqn, enable_dueling_network, dueling_type, *args, **kwargs)
    105 super(DQNAgent, self).__init__(*args, **kwargs)
    107 # Validate (important) input.
--> 108 if hasattr(model.output, '__len__') and len(model.output) > 1:
    109     raise ValueError('Model "{}" has more than one output. DQN expects a model that has a single output.'.format(model))
    110 if model.output._keras_shape != (None, self.nb_actions):

File ~/miniconda3/envs/hf/lib/python3.10/site-packages/keras/engine/keras_tensor.py:221, in KerasTensor.__len__(self)
    220 def __len__(self):
--> 221   raise TypeError('Keras symbolic inputs/outputs do not '
    222                   'implement `__len__`. You may be '
    223                   'trying to pass Keras symbolic inputs/outputs '
    224                   'to a TF API that does not register dispatching, '
    225                   'preventing Keras from automatically '
    226                   'converting the API call to a lambda layer '
    227                   'in the Functional Model. This error will also get raised '
    228                   'if you try asserting a symbolic input/output directly.')

TypeError: Keras symbolic inputs/outputs do not implement `__len__`. You may be trying to pass Keras symbolic inputs/outputs to a TF API that does not register dispatching, preventing Keras from automatically converting the API call to a lambda layer in the Functional Model. This error will also get raised if you try asserting a symbolic input/output directly.
In [ ]:
In [74]:
cv2.imread("training_set\Gray Wagtail\ML169066401.png")

[ WARN:0@5159.597] global loadsave.cpp:244 findDecoder imread_('training_set\Gray Wagtail\ML169066401.png'): can't open/read file: check file path/integrity
In [76]:
import cv2

def preprocess_image(image_path):
    try:
        # Load the image from the specified path
        image = cv2.imread(image_path)

        # Check if the image is None or empty
        if image is None or image.size == 0:
            raise ValueError(f"Failed to read image from {image_path}")

        # Resize the image to a fixed size
        image = cv2.resize(image, (224, 224))

        # Convert the image to floating point format
        image = image.astype('float32')

        # Normalize the image by dividing by 255
        image /= 255

        return image
    except Exception as e:
        print(image_path)
        print(f"Error processing image {image_path}: {str(e)}")
        return None
In [88]:
# Define a custom gym environment
class CustomEnv(gym.Env):
    def __init__(self):
        self.train_dir = "training_set"
        self.test_dir = "testing_set"
        self.training_mode = True
        self.observation_space = gym.spaces.Box(
            low=0,
            high=255,
            shape=(224, 224, 3),
            dtype='uint8'
        )
        self.action_space = gym.spaces.Box(
            low=-1,
            high=1,
            shape=(1,),
            dtype='float32'
        )
        # Initialize ResNet50V2 model
        self.model = model
        # Unfreeze last 5 layers for fine-tuning
        for layer in self.model.layers[-5:]:
            layer.trainable = True
        # Add top layers for action prediction
        x = Flatten()(self.model.output)
        x = Dense(512, activation='relu')(x)
        self.output_layer = Dense(self.action_space.shape[0])(x)
        self.model = Model(inputs=self.model.input, outputs=self.output_layer)
        # Define optimizer
        self.optimizer = Adam(learning_rate=0.001)
        # Define loss function
        self.loss_fn = tf.keras.losses.MeanSquaredError()
        self.current_observation = None
        self.current_step = 0
        self.max_steps = 1000
        self.reward_range = (-float('inf'), float('inf'))
        
    def _get_observation(self):
        if self.training_mode:
            # Get the path to a random image in the train directory
            class_dir = np.random.choice(os.listdir(self.train_dir))
            image_file = os.path.join(self.train_dir, class_dir, np.random.choice(os.listdir(os.path.join(train_dir, class_dir))))
        else:
            # Get the path to the next image in the test directory
            class_dir = np.random.choice(os.listdir(self.test_dir))
            image_file = os.path.join(self.test_dir, class_dir, np.random.choice(os.listdir(os.path.join(test_dir, class_dir))))

        # Preprocess the image
        image = preprocess_image(image_file)

        # Get the current joint angles and velocities

        # Combine the image and joint states into an observation
        observation = {"image": image}

        return observation

    def reset(self):
        self.current_step = 0
        
        # Generate a new observation as the initial state
        observation = self._get_observation()
        self.current_observation = observation
        
        # Return the initial observation
        return observation

    def step(self, action):
        # Take action in environment and return new observation, reward, and done flag
        self.current_step += 1
        new_state = self.current_observation + action
        features = self.model.predict(new_state[None])[0]
        reward = self._compute_reward(new_state, self.current_observation, action)
        done = (self.current_step >= self.max_steps)
        self.current_observation = new_state
        return features.reshape(-1), reward, done, {}
    
    def _compute_reward(self, new_state, previous_state, action):
        # Compute the reward based on the new and previous states and the action taken
        reward = np.sum(np.abs(new_state - previous_state))

        # Return the reward
        return reward

    def render(self, mode='human'):
        # Render environment
        pass
In [107]:
# Define RL agent
class Agent:
    def __init__(self, env):
        self.env = env
        self.observation_space = env.observation_space
        self.action_space = env.action_space
        self.policy_network = self._build_policy_network()

    def _build_policy_network(self):
        inputs = tf.keras.layers.Input(shape=self.observation_space.shape)
        # x = tf.keras.applications.resnet_v2.preprocess_input(inputs)
        resnet = model
        for layer in resnet.layers[:-5]:
            layer.trainable = False
        x = resnet(inputs)
        x = tf.keras.layers.Flatten()(x)
        x = tf.keras.layers.Dense(64, activation='relu')(x)
        outputs = tf.keras.layers.Dense(self.action_space.shape[0], activation='tanh')(x)
        policy_network = tf.keras.models.Model(inputs=inputs, outputs=outputs)
        return policy_network
    
    def choose_action(self, observation):
        # Use the policy network to generate an action
        action = self.policy_network.predict([state])[0]
        
        # Clip the action value to be within the valid range
        action = np.clip(action, self.action_space.low, self.action_space.high)

        # Return the action
        return action

    def train(self, iterations):
        for i in range(iterations):
            observation = self.env.reset()
            done = False
            while not done:
                action = self.choose_action(observation)
                new_observation, reward, done, _ = self.env.step(action)
                # Update policy network using RL algorithm
                pass
In [108]:
import gym
import numpy as np

# Create an instance of the CustomEnv class
env = CustomEnv()

# Create an instance of the Agent class
agent = Agent(env)

# Define the training loop
num_episodes = 1000
max_steps_per_episode = 1000
batch_size = 16
discount_factor = 0.99
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)

for episode in range(num_episodes):
    # Reset the environment for the start of the episode
    state = env.reset()
    episode_reward = 0
    episode_states = []
    episode_actions = []
    episode_rewards = []

    for step in range(max_steps_per_episode):
        # Choose an action based on the current state
        action = agent.choose_action(state)

        # Take a step in the environment based on the chosen action
        new_state, reward, done, info = env.step(action)
        
        # Compute the reward for the current step
        previous_state = state if step == 0 else episode_states[-1]
        step_reward = agent._compute_reward(new_state, previous_state, action)

        # Add the state, action, and reward to the episode history
        episode_states.append(state)
        episode_actions.append(action)
        episode_rewards.append(step_reward)
        episode_reward += step_reward

        # Update the current state
        state = new_state

        # If the episode is over, break out of the loop
        if done:
            break

    # Compute the discounted rewards for each step in the episode
    episode_discounted_rewards = np.zeros_like(episode_rewards)
    cumulative_reward = 0
    for i in reversed(range(len(episode_rewards))):
        cumulative_reward = episode_rewards[i] + discount_factor * cumulative_reward
        episode_discounted_rewards[i] = cumulative_reward

    # Normalize the discounted rewards
    episode_discounted_rewards -= np.mean(episode_discounted_rewards)
    episode_discounted_rewards /= np.std(episode_discounted_rewards)

    # Train the policy network using the episode history
    num_batches = len(episode_states) // batch_size
    for batch in range(num_batches):
        batch_start = batch * batch_size
        batch_end = (batch + 1) * batch_size

        # Get the batch of states, actions, and discounted rewards
        batch_states = np.array(episode_states[batch_start:batch_end])
        batch_actions = np.array(episode_actions[batch_start:batch_end])
        batch_discounted_rewards = np.array(episode_discounted_rewards[batch_start:batch_end])

        # Compute the loss and gradients
        with tf.GradientTape() as tape:
            logits = agent.policy_network(batch_states)
            loss = -tf.reduce_mean(batch_discounted_rewards * tf.math.log(logits + 1e-10) + (1 - batch_discounted_rewards) * tf.math.log(1 - logits + 1e-10))
        gradients = tape.gradient(loss, agent.policy_network.trainable_variables)

        # Apply the gradients to the optimizer
        optimizer.apply_gradients(zip(gradients, agent.policy_network.trainable_variables))

    # Print the episode reward every 10 episodes
    if (episode + 1) % 10 == 0:
        print(f"Episode {episode + 1}, Reward: {episode_reward}")

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[110], line 27
     23 episode_rewards = []
     25 for step in range(max_steps_per_episode):
     26     # Choose an action based on the current state
---> 27     action = agent.choose_action(state)
     29     # Take a step in the environment based on the chosen action
     30     new_state, reward, done, info = env.step(action)

Cell In[109], line 24, in Agent.choose_action(self, observation)
     22 def choose_action(self, observation):
     23     # Use the policy network to generate an action
---> 24     action = self.policy_network.predict([state])[0]
     26     # Clip the action value to be within the valid range
     27     action = np.clip(action, self.action_space.low, self.action_space.high)

File ~/miniconda3/envs/hf/lib/python3.10/site-packages/keras/utils/traceback_utils.py:67, in filter_traceback.<locals>.error_handler(*args, **kwargs)
     65 except Exception as e:  # pylint: disable=broad-except
     66   filtered_tb = _process_traceback_frames(e.__traceback__)
---> 67   raise e.with_traceback(filtered_tb) from None
     68 finally:
     69   del filtered_tb

File /tmp/__autograph_generated_filegpzrq7x9.py:15, in outer_factory.<locals>.inner_factory.<locals>.tf__predict_function(iterator)
     13 try:
     14     do_return = True
---> 15     retval_ = ag__.converted_call(ag__.ld(step_function), (ag__.ld(self), ag__.ld(iterator)), None, fscope)
     16 except:
     17     do_return = False

ValueError: in user code:

    File "/home/priteshk/miniconda3/envs/hf/lib/python3.10/site-packages/keras/engine/training.py", line 1845, in predict_function  *
        return step_function(self, iterator)
    File "/home/priteshk/miniconda3/envs/hf/lib/python3.10/site-packages/keras/engine/training.py", line 1834, in step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/priteshk/miniconda3/envs/hf/lib/python3.10/site-packages/keras/engine/training.py", line 1823, in run_step  **
        outputs = model.predict_step(data)
    File "/home/priteshk/miniconda3/envs/hf/lib/python3.10/site-packages/keras/engine/training.py", line 1791, in predict_step
        return self(x, training=False)
    File "/home/priteshk/miniconda3/envs/hf/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
        raise e.with_traceback(filtered_tb) from None
    File "/home/priteshk/miniconda3/envs/hf/lib/python3.10/site-packages/keras/engine/input_spec.py", line 264, in assert_input_compatibility
        raise ValueError(f'Input {input_index} of layer "{layer_name}" is '

    ValueError: Input 0 of layer "model_57" is incompatible with the layer: expected shape=(None, 224, 224, 3), found shape=(32, 224, 3)
In [ ]:

Predict on Test Model¶

In [9]:
preds_probs = model.predict(test_data)

2023-03-28 16:24:20.320572: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8700

  5/353 [..............................] - ETA: 13s 
2023-03-28 16:24:24.101015: I tensorflow/stream_executor/cuda/cuda_blas.cc:1786] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.

239/353 [===================>..........] - ETA: 4s
2023-03-28 16:24:33.165492: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

251/353 [====================>.........] - ETA: 3s
2023-03-28 16:24:33.585175: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-28 16:24:33.585680: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

318/353 [==========================>...] - ETA: 1s
2023-03-28 16:24:36.221128: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

353/353 [==============================] - 23s 41ms/step
In [10]:
preds_probs
Out[10]:
array([[5.57500064e-01, 1.49156470e-02, 3.95028167e-07, ...,
        4.03020772e-09, 1.92685297e-06, 7.63295027e-09],
       [8.00372899e-01, 6.03676625e-02, 9.12816540e-05, ...,
        4.56398884e-05, 1.27164649e-05, 1.05896497e-05],
       [9.99792397e-01, 1.87173882e-05, 1.92202324e-10, ...,
        1.31564887e-12, 1.08445420e-04, 1.29400293e-11],
       ...,
       [3.80474596e-13, 2.90768583e-08, 4.39055185e-07, ...,
        3.23493628e-11, 5.69656819e-08, 9.99963522e-01],
       [1.64871852e-10, 4.19621711e-06, 4.79005530e-06, ...,
        1.58595073e-03, 1.12237494e-07, 9.04017329e-01],
       [3.26277561e-10, 1.61820171e-05, 5.30638536e-05, ...,
        1.70158764e-05, 3.15923759e-10, 9.24969256e-01]], dtype=float32)
In [11]:
prediction = np.argmax(preds_probs, axis = 1)
prediction
Out[11]:
array([ 0,  0,  0, ..., 24, 24, 24])

Evaluate on the Test Set¶

In [12]:
test_loss, test_accuracy = model.evaluate(test_data, verbose=0)
print("Loss on test set: {}".format(test_loss))
print("Accuracy on test set: {:.4f}%".format(test_accuracy*100))

2023-03-28 16:24:57.769452: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-28 16:24:58.232488: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-28 16:24:58.232669: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-28 16:25:01.064482: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile

Loss on test set: 0.4836946427822113
Accuracy on test set: 86.9488%

Make Confusion Matrix¶

In [13]:
from helper_functions import make_confusion_matrix
In [14]:
# To get out test labels we need to unravel out test_data BatchDataset
y_label = []
for images, labels in test_data.unbatch():
    y_label.append(labels.numpy().argmax()) # currently test labels look like: [0, 0, 0, 0, 1 ... , 0, 0]
    
y_label[:10] # look at first 10

2023-03-28 16:25:07.967853: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-28 16:25:08.195947: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-28 16:25:08.203304: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
2023-03-28 16:25:09.665524: W tensorflow/core/lib/png/png_io.cc:88] PNG warning: iCCP: known incorrect sRGB profile
Out[14]:
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
In [15]:
len(y_label)
Out[15]:
5647
In [16]:
make_confusion_matrix(y_true=y_label, y_pred=prediction, classes=test_data.class_names, figsize=(100, 100))

Creating Classification Report¶

In [17]:
from sklearn.metrics import classification_report
print(classification_report(y_true=y_label, y_pred=prediction))

              precision    recall  f1-score   support

           0       0.86      0.90      0.88       231
           1       0.76      0.64      0.70       232
           2       0.96      0.98      0.97       229
           3       0.94      0.92      0.93       231
           4       0.88      0.94      0.91       231
           5       0.73      0.80      0.76       229
           6       0.90      0.75      0.82       229
           7       0.89      0.69      0.78       194
           8       0.81      0.90      0.85       231
           9       0.82      0.88      0.85       231
          10       0.96      0.92      0.94       231
          11       0.88      0.88      0.88       231
          12       0.76      0.85      0.80       230
          13       0.93      0.92      0.92       229
          14       0.86      0.87      0.87       157
          15       0.95      0.82      0.88       231
          16       0.73      0.89      0.80       232
          17       0.90      0.89      0.89       232
          18       0.89      0.88      0.88       232
          19       0.95      0.90      0.92       231
          20       0.75      0.89      0.81       215
          21       0.96      0.91      0.94       232
          22       0.94      0.90      0.92       232
          23       0.93      0.91      0.92       232
          24       0.92      0.91      0.91       232

    accuracy                           0.87      5647
   macro avg       0.87      0.87      0.87      5647
weighted avg       0.87      0.87      0.87      5647
In [18]:
# Get a dictionary of classification report
classification_report_dict = classification_report(y_true=y_label, y_pred=prediction, output_dict=True)
In [19]:
class_names = test_data.class_names

Plot F1 Scores¶

In [33]:
# Create empty dictionary
class_f1_scores = {}
accuracy = None

# Loop through classification report
for k, v in classification_report_dict.items():
    if k == "accuracy": # Stop once we get to accuracy key
        accuracy = v
        break
    else:
        # Add class names and F1 scores o new dictionaty
        class_f1_scores[class_names[int(k)]] = v["f1-score"]
class_f1_scores
Out[33]:
{'Asian Green Bee-Eater': 0.8752642706131079,
 'Brown-Headed Barbet': 0.6962616822429907,
 'Cattle Egret': 0.9696969696969697,
 'Common Kingfisher': 0.9301310043668122,
 'Common Myna': 0.9056603773584906,
 'Common Rosefinch': 0.760914760914761,
 'Common Tailorbird': 0.8162291169451074,
 'Coppersmith Barbet': 0.7790697674418604,
 'Forest Wagtail': 0.8536082474226804,
 'Gray Wagtail': 0.8458333333333333,
 'Hoopoe': 0.9380530973451326,
 'House Crow': 0.8806941431670282,
 'Indian Grey Hornbill': 0.8049281314168377,
 'Indian Peacock': 0.9210526315789475,
 'Indian Pitta': 0.8670886075949368,
 'Indian Roller': 0.8796296296296297,
 'Jungle Babbler': 0.8015564202334631,
 'Northern Lapwing': 0.8917748917748919,
 'Red-Wattled Lapwing': 0.8845315904139432,
 'Ruddy Shelduck': 0.9247787610619468,
 'Rufous Treepie': 0.8110403397027601,
 'Sarus Crane': 0.9380530973451328,
 'White Wagtail': 0.9183222958057395,
 'White-Breasted Kingfisher': 0.9150326797385621,
 'White-Breasted Waterhen': 0.9110629067245121}
In [32]:
# turn f1 scores into dataframe for visualization
import pandas as pd

f1_scores = pd.DataFrame({"class_names": list(class_f1_scores.keys()),
                         "f1-score": list(class_f1_scores.values())}).sort_values("f1-score", ascending=False)
In [38]:
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(25, 8))
scores = ax.bar(range(len(f1_scores)), f1_scores["f1-score"].values)
ax.set_xticks(range(len(f1_scores)))
ax.set_xticklabels(f1_scores["class_names"], rotation=90, ha='right')
ax.set_ylabel("F1-score")
plt.axhline(y=accuracy, linewidth=2, color='r')

ax.set_title("F1-score for 25 Bird classes with Accuracy Threshold line")
ax.bar_label(scores, fmt="%.2f");

Visualizing predictions on test images¶

Steps:

  • Read in a target inmage filepath using tf.io.read_file()
  • turn the image into a tensor using tf.io.decode_image()
  • Resize the image tensor to be the same size as images our model has trained on using tf.image.resize()
  • Scale the image to get all of the pizel values between 0 & 1
In [62]:
# Create a function to load and prepare images
def load_and_prep_image(filename, img_size=224, scale=True):
    """
    reads in an image form filename, turns it into a tensor and reshapes into specified shape
    
    Args:
        filename (str): path to target image
        img_shape (int): height/width dimentions of target image size
        scale (bool): scale pixel values from 0-255 to  0-1 or not
        
    Returns:
        Image tensor of shape (img_shape, img_shape, 3)
    """
    # Read in the image
    img = tf.io.read_file(filename=filename)
    
    # Decode image into tensor
    img = tf.io.decode_image(img, channels=3)
    
    # Resize the image
    img = tf.image.resize(img, [img_size, img_size])
    
    # Scaling
    if scale:
        # Rescale the image
        return img/255.
    else:
        return img
In [63]:
# visualize test images

# Make preds on series of random images
import os
import random

plt.figure(figsize=(17, 10))
for i in range(3):
    # Choose a random image from random class
    class_name = random.choice(class_names)
    filename = random.choice(os.listdir(test_dir + "/" + class_name))
    filepath = test_dir + "/" + class_name + "/" + filename
    
    # Load the images and make predictions
    img = load_and_prep_image(filepath, scale=False)
    img_expanded = tf.expand_dims(img, axis=0)
    # pred_prob = model.predict(img_expanded) # get perdiction probability array
    pred_prob = model.predict(img_expanded) # get perdiction probability array
    pred_class = class_names[pred_prob.argmax()] # get highest prediction probability index
    
    # Plot the images
    plt.subplot(1, 3, i+1)
    plt.imshow(img/255)
    if class_name == pred_class: # if predicted class matches, truth class, make text green
        title_color = "g"
    else:
        title_color = "r"
    plt.title(f"Actual class: {class_name}, pred: {pred_class}, prob: {pred_prob.max():.2f}", c=title_color)
    plt.axis(False);    

1/1 [==============================] - 0s 106ms/step
1/1 [==============================] - 0s 26ms/step
1/1 [==============================] - 0s 30ms/step

Finding the most wrong predictions¶

In [64]:
# get all of the files from test_data
filepaths = []
for filepath in test_data.list_files("testing_set/*/*.jpg", shuffle=False):
    filepaths.append(filepath.numpy())
    
filepaths[:10]
Out[64]:
[b'testing_set/Asian Green Bee-Eater/ML100845971.jpg',
 b'testing_set/Asian Green Bee-Eater/ML102065651.jpg',
 b'testing_set/Asian Green Bee-Eater/ML108389751.jpg',
 b'testing_set/Asian Green Bee-Eater/ML108490471.jpg',
 b'testing_set/Asian Green Bee-Eater/ML109116691.jpg',
 b'testing_set/Asian Green Bee-Eater/ML112055301.jpg',
 b'testing_set/Asian Green Bee-Eater/ML118156651.jpg',
 b'testing_set/Asian Green Bee-Eater/ML118645851.jpg',
 b'testing_set/Asian Green Bee-Eater/ML119513941.jpg',
 b'testing_set/Asian Green Bee-Eater/ML120026161.jpg']
In [65]:
# Create a dataframe of different parameters
pred_df = pd.DataFrame({"img_path": filepaths,
                       "y_true": y_label,
                       "y_pred": prediction,
                       "pred_conf": preds_probs.max(axis=1), # Get the maximum prediction probability value
                       "y_true_classname": [class_names[i] for i in y_label],
                       "y_pred_classname": [class_names[i] for i in prediction]})
pred_df
Out[65]:
img_path y_true y_pred pred_conf y_true_classname y_pred_classname
0 b'testing_set/Asian Green Bee-Eater/ML10084597... 0 0 0.966927 Asian Green Bee-Eater Asian Green Bee-Eater
1 b'testing_set/Asian Green Bee-Eater/ML10206565... 0 0 0.999225 Asian Green Bee-Eater Asian Green Bee-Eater
2 b'testing_set/Asian Green Bee-Eater/ML10838975... 0 12 0.334575 Asian Green Bee-Eater Indian Grey Hornbill
3 b'testing_set/Asian Green Bee-Eater/ML10849047... 0 0 0.999982 Asian Green Bee-Eater Asian Green Bee-Eater
4 b'testing_set/Asian Green Bee-Eater/ML10911669... 0 0 0.958633 Asian Green Bee-Eater Asian Green Bee-Eater
... ... ... ... ... ... ...
4509 b'testing_set/White-Breasted Waterhen/ML862254... 24 24 0.725891 White-Breasted Waterhen White-Breasted Waterhen
4510 b'testing_set/White-Breasted Waterhen/ML870705... 24 24 0.999892 White-Breasted Waterhen White-Breasted Waterhen
4511 b'testing_set/White-Breasted Waterhen/ML872301... 24 24 0.999891 White-Breasted Waterhen White-Breasted Waterhen
4512 b'testing_set/White-Breasted Waterhen/ML912482... 24 24 0.970743 White-Breasted Waterhen White-Breasted Waterhen
4513 b'testing_set/White-Breasted Waterhen/ML923868... 24 24 0.999579 White-Breasted Waterhen White-Breasted Waterhen

4514 rows × 6 columns

In [66]:
# Find which predictions are wrong
pred_df["pred_correct"] = pred_df["y_true"] == pred_df["y_pred"]
pred_df.head()
Out[66]:
img_path y_true y_pred pred_conf y_true_classname y_pred_classname pred_correct
0 b'testing_set/Asian Green Bee-Eater/ML10084597... 0 0 0.966927 Asian Green Bee-Eater Asian Green Bee-Eater True
1 b'testing_set/Asian Green Bee-Eater/ML10206565... 0 0 0.999225 Asian Green Bee-Eater Asian Green Bee-Eater True
2 b'testing_set/Asian Green Bee-Eater/ML10838975... 0 12 0.334575 Asian Green Bee-Eater Indian Grey Hornbill False
3 b'testing_set/Asian Green Bee-Eater/ML10849047... 0 0 0.999982 Asian Green Bee-Eater Asian Green Bee-Eater True
4 b'testing_set/Asian Green Bee-Eater/ML10911669... 0 0 0.958633 Asian Green Bee-Eater Asian Green Bee-Eater True
In [67]:
# Sort dataframe to have most wrong predictions at top
top_100_wrong = pred_df[pred_df["pred_correct"] == False].sort_values("pred_conf", ascending=False)[:100]
top_100_wrong.head(20)
Out[67]:
img_path y_true y_pred pred_conf y_true_classname y_pred_classname pred_correct
191 b'testing_set/Brown-Headed Barbet/ML119679471.... 1 7 0.999370 Brown-Headed Barbet Coppersmith Barbet False
2099 b'testing_set/House Crow/ML256308541.jpg' 11 4 0.999160 House Crow Common Myna False
1135 b'testing_set/Common Tailorbird/ML149453221.jpg' 6 9 0.998646 Common Tailorbird Gray Wagtail False
2331 b'testing_set/Indian Grey Hornbill/ML63915191.... 12 8 0.998476 Indian Grey Hornbill Forest Wagtail False
565 b'testing_set/Common Kingfisher/ML152225741.jpg' 3 23 0.998271 Common Kingfisher White-Breasted Kingfisher False
231 b'testing_set/Brown-Headed Barbet/ML196420071.... 1 16 0.996154 Brown-Headed Barbet Jungle Babbler False
2917 b'testing_set/Jungle Babbler/ML282513391.jpg' 16 6 0.995713 Jungle Babbler Common Tailorbird False
4169 b'testing_set/White-Breasted Kingfisher/ML1646... 23 18 0.995000 White-Breasted Kingfisher Red-Wattled Lapwing False
806 b'testing_set/Common Myna/ML221310241.jpg' 4 24 0.994229 Common Myna White-Breasted Waterhen False
1338 b'testing_set/Coppersmith Barbet/ML176468941.jpg' 7 11 0.993855 Coppersmith Barbet House Crow False
1306 b'testing_set/Coppersmith Barbet/ML131700641.jpg' 7 14 0.993511 Coppersmith Barbet Indian Pitta False
3887 b'testing_set/Sarus Crane/ML350856521.jpg' 21 2 0.990375 Sarus Crane Cattle Egret False
4109 b'testing_set/White Wagtail/ML498687651.jpg' 22 8 0.990031 White Wagtail Forest Wagtail False
361 b'testing_set/Brown-Headed Barbet/ML91802381.jpg' 1 5 0.988294 Brown-Headed Barbet Common Rosefinch False
1279 b'testing_set/Common Tailorbird/ML81698581.jpg' 6 16 0.984620 Common Tailorbird Jungle Babbler False
1492 b'testing_set/Forest Wagtail/ML212905501.jpg' 8 9 0.983442 Forest Wagtail Gray Wagtail False
2683 b'testing_set/Indian Roller/ML120308811.jpg' 15 5 0.980947 Indian Roller Common Rosefinch False
4469 b'testing_set/White-Breasted Waterhen/ML351459... 24 20 0.979722 White-Breasted Waterhen Rufous Treepie False
1331 b'testing_set/Coppersmith Barbet/ML155867261.jpg' 7 20 0.979433 Coppersmith Barbet Rufous Treepie False
1190 b'testing_set/Common Tailorbird/ML205140771.jpg' 6 9 0.975412 Common Tailorbird Gray Wagtail False
In [68]:
# Visualize wrong predictions
images_to_view = 9
start_index = 10
plt.figure(figsize=(30, 20))
for i, row in enumerate(top_100_wrong[start_index:start_index + images_to_view].itertuples()):
    plt.subplot(3, 3, i+1)
    img = load_and_prep_image(row[1], scale=False)
    _, _, _, _, pred_prob, y_true_classname, y_pred_classname, _ = row # skipping few values
    plt.imshow(img/255)
    plt.title(f"actual: {y_true_classname}, pred: {y_pred_classname}, prob: {pred_prob}")
    plt.axis(False);

Testing on Custom Images¶

In [74]:
custom_bird_images = ["custom_images/" + img_path for img_path in os.listdir("custom_images")]
In [75]:
custom_bird_images
Out[75]:
['custom_images/hoopoe.jpg',
 'custom_images/Indian Pitta.jpg',
 'custom_images/crow.jpg',
 'custom_images/Common tailorbird Orthotomus sutorius.jpg',
 'custom_images/Common myna.jpg']
In [76]:
# make predictions and plot custom food images
for img in custom_bird_images:
    img = load_and_prep_image(img, scale=False)
    pred_prob = model.predict(tf.expand_dims(img, axis=0))
    pred_class = class_names[pred_prob.argmax()]
    
    # Plot the information
    plt.figure()
    plt.imshow(img/255.)
    plt.title(f"pred: {pred_class}, prob: {pred_prob.max():.2f}")
    plt.axis(False);

1/1 [==============================] - 0s 108ms/step
1/1 [==============================] - 0s 29ms/step
1/1 [==============================] - 0s 65ms/step
1/1 [==============================] - 0s 24ms/step
1/1 [==============================] - 0s 26ms/step

Using TensorBoard for better visualizations¶

In [128]:
%load_ext tensorboard
In [137]:
!tensorboard dev auth revoke

2023-03-28 16:08:21.014981: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-03-28 16:08:22.259226: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:08:22.312230: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:08:22.312308: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
Logged out of uploader.
In [ ]:
!tensorboard dev upload --one_shot --logdir transfer_learning/

2023-03-28 16:08:26.376638: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-03-28 16:08:27.540871: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:08:27.589219: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.
2023-03-28 16:08:27.589297: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:961] could not open file to read NUMA node: /sys/bus/pci/devices/0000:01:00.0/numa_node
Your kernel may have been built without NUMA support.

***** TensorBoard Uploader *****

This will upload your TensorBoard logs to https://tensorboard.dev/ from
the following directory:

transfer_learning/

This TensorBoard will be visible to everyone. Do not upload sensitive
data.

Your use of this service is subject to Google's Terms of Service
<https://policies.google.com/terms> and Privacy Policy
<https://policies.google.com/privacy>, and TensorBoard.dev's Terms of Service
<https://tensorboard.dev/policy/terms/>.

This notice will not be shown again while you are logged into the uploader.
To log out, run `tensorboard dev auth revoke`.

Continue? (yes/NO)

TensorBoard Experiment: https://tensorboard.dev/experiment/dX0p7nQQRPGpFxozjjaZkw/#scalars&runSelectionState=eyJiYXNlX21vZGVsLzIwMjMwMzI1LTE5NDEwNC90cmFpbiI6ZmFsc2V9&run=base_model%2F20230325-193754%2Ftrain

In [ ]: